A Connection Between Variable Selection and EM-Type Algorithms

Variable selection is fundamental to high-dimensional statistical modeling. Fan and Li (2001) proposed a class of variable selection procedures via nonconcave penalized likelihood. Optimizing the penalized likelihood function is challenging as it is a highdimensional nonconcave function with singularities. A new algorithm is proposed for finding a solution of the nonconcave penalized likelihood via a modified local quadratic approximation. The proposed algorithm repairs the drawback of Fan and Li’s algorithm. We establish a connection between local quadratic approximation and the so-called MM algorithms, useful extensions of the EM algorithms. This connection enables us to analyze the local and global convergence of the local quadratic approximation algorithm by employing the techniques used for EM algorithms. Moreover, this connection provides a general scheme for constructing a minorizing function in the MM algorithm via the local quadratic approximation.